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1. 


INTRODUCTION. 


1.1  BACKGROUND. 

a.  An  increase  in  threat  conditions  could  result  in  a  requirement  for  additional  security 
precautions  such  as  a  positive  passenger-bag  match  or  x-ray  inspectioii  of  all  baggage.  These  precautions 
are  resource  intensive;  however,  their  cost  could  be  substantially  reduced  if  the  number  of  passengers 
needing  special  security  measures  could  be  minimized.  This  could  potentially  be  accomplished  by 
applying  a  passenger  "profile"  to  identify  domestic  travelers  thought  not  to  represent  a  terrorist  threat. 

b.  Profiles  exist  for  detecting  potential  hijackers  as  well  as  terrorists  on  international  flights. 
International  profiling  includes  the  Risk  Assessment  Profile  System  (RAPS)  and  Northwest  Airlines  has 
an  experimental  version  of  an  Automated  Profiling  System  (APS)  in  development  for  international 
passengers.  The  approach  being  taken  for  domestic  passengers  is  to  clear  those  judged  not  to  be  a  threat 
as  opposed  to  identifying  threat  individuals  or  "selectees.” 

c.  The  Office  of  Civil  Aviation  Security  Policy  and  Planning  (ACP)  and  Office  of  Civil 
Aviation  Security  Intelligence  (ACI)  Federal  Aviation  Administration  (FAA)  headquarters  elements  of 
Assistant  Administration  for  Civil  Aviation  Security  (ACS)  require  the  design  and  feasibility  analysis 
of  a  method  for  manually  profiling  domestic  passengers.  The  present  effort  focuses  on  profiling  which 
is  passive,  meaning  that  passengers  are  not  directly  queried.  The  anticipated  technique  will  feature 
information  contained  in  the  Customer  Reservation  System  (CRS)  as  well  as  any  other  relevant 
information  readily  available  to  the  profiling  airline  agent.  Scoring  or  profiling  will  consist  of  a 
mathematical  score  resulting  from  analysis  of  the  combined  data  elements  and  their  individual  weighting 
factors. 


d.  The  focus  will  be  on  using  the  profile  score  to  eliminate  low-risk  passengers  from 
additional  special  scrutiny  (e.g.,  baggage  matching,  extensive  baggage  inspection).  An  automated  version 
of  the  profiling  procedure  developed  is  anticipated  for  future  development. 

e.  This  research  is  conducted  by  the  Aviation  Security  Human  Factors  Program  under 
Research  Project  Initiative  (RPI)  #129  in  suppon  of  Mission  Need  Statement  (MNS)  #163. 

1.2  SYSTEM  DESCRIPTION. 

The  Manual  Domestic  Passive  Profiling  System  (MDPPS)  being  evaluated  emphasizes  input  data  from 
Northwest’s  Passenger  Name  Record  (PNR).  This  reservation  system  is  also  used  by  Delta  and  TWA. 
Data  will  be  entered  on  a  paper  and  pencil  instrument  (evaluation/rating  sheet)  to  be  evaluated  by  the 
specified  methodology  developed.  The  outcome  will  be  a  MDPPS  score  which  identifies  passengers  on 
a  domestic  flight  who  do  not  need  additional  special  screening.  Those  not  eliminated  are  candidates  for 
a  positive  baggage  match  and/or  interrogatory  intervention.  If  a  useful  MDPPS  is  developed,  future 
work  will  be  needed  to  apply  the  technique  to  the  CRS  for  the  remaining  major  air  carriers  such  as 
United  (APOLLO  CRS)  and  American  (SABER  CRS). 
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1.3  CRITICAL  OPERATIONAL  ISSUES  AND  CRITERIA  (COICV 

Three  critical  operational  issues  will  be  tested  and  evaluated.  Section  2  describes  theu^  scope,  criteria, 
rationale,  evaluation  approach,  analysis  of  Measures  of  Performance  (MOP),  and  data  presentation. 

1.3.1  Issue  1. 

Do  airlines’  CRS  databases  contain  information  elements  appropriate  for  judging  passengers  not  to  be 
a  threat  on  domestic  flights? 

1.3.2  Issue  2. 

Does  the  profiling  system  eliminate  most  domestic  passengers  from  requiring  special  security  treatment? 

1.3.3  Issue  3. 

Does  the  domestic  passenger  profiling  system  require  excessive  resource  requirements? 

1.4  TEST  AND  EVALUATION  MILESTONES. 

Milestones  have  been  established  in  table  1  to  ensure  orderly  execution  of  the  Test  and  Evaluation 
(T&E)  process  in  terms  of  planning,  programming  and  reporting. 

TABLE  1.  TEST  AND  EVALUATION  MILESTONES 


Event 


Completion 

Schedule 


Phase  I: 

Task  1:  Develop  Test  and  Evaluation  Plan  (TEP)  21  Jan  94 

Complete  draft  of  Government  Furnished  Information 
(GFI)  TEP  format 

Incorporate  FAA  review  and  comments 

Task  2:  Form  Subject-Matter  Expert  (SME)  panel  30  Jan  94 

Develop  FAA  membership  list  for  SME  panel 
Finalize  SME  panel  and  workshop  date(s) 

Task  3:  Panel  evaluation  of  Northwest  information  elements  14  Feb  94 

and  rating  factors  for  in  MDPPS 
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TABLE  1.  TEST  AND  EVALUATION  MILESTONES  (Continued) 


Event 

Completion 

Schedule 

Obtain  PNR  information  elements 

Develop  SME  workshop  materials  and  protocols 

Conduct  workshop  to  review  GFI  strawman  profiling  system 
and  develop  prototype  MDPPS  instrument 

Phase  II: 

Task  4:  Conduct  Operational  Test  and  Evaluation  (OT&E) 

25  Feb  94 

Train  Northwest  employees  to  use  the  MDPPS 

Conduct  pilot  test 

Gather  TEP-specified  data  at  Milwaukee  Airport/NW 

Task  5:  Analyze  data 

28  Feb  94 

Determine  percentage  of  passengers  cleared 

Determine  time  requirements  for  using  MDPPS 

Estimate  cost  effectiveness 

Task  6;  Draft  Test  and  Evaluation  Report  (TER) 

10  Mar  94 

Complete  draft  of  GFI  TER  format 

Incorporate  FAA  review  and  comments 

Task  7;  Final  TER 

31  Mar  94 
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2. 


TEST  AND  EVALUATION  STRATEGY. 


2.1  OPERATIONAL  EFFECTIVENESS. 

The  operational  effectiveness  of  passive  domestic  profiling  is  addressed  through  critical  operational 
issues  on  objective  profiling  criteria  and  percentage  of  passengers  cleared. 

2.1.1  Issue  1  -  Profiling  Criteria. 

Do  airlines’  CRS  databases  contain  information  elements  appropriate  for  identifying  passengers  judged 
not  to  be  a  threat  on  domestic  flights? 

2. 1 . 1 . 1  Scope. 

Airline  reservations  systems  contain  data  fields  for  information  such  as  mode  of  payment,  routing, 
frequent  flyer  membership,  etcetera.  Table  2  presents  the  information  from  Northwest’s  PNR  that  will 
be  given  to  a  panel  of  SMEs.  They  will  also  be  provided  with  a  list  of  reservation  information  proposed 
by  (ACI)  for  domestic  profiling  (table  3).  The  SMEs  will  evaluate  information  readily  available  to 
airline  personnel  and  judge  whether  it  can  reasonably  be  expected  to  identify  domestic  passengers  not 
needing  special  security  measures. 

2. 1.1.2  Criteria. 

a.  The  SME  panel  (table  4)  will  recommend  particular  information  items  lo  be  used  as 
factors  in  the  MDPPS  job  aid. 

b.  The  SME  panel  will  recommend  the  weighting  of  information  items  to  be  used  as  factors 
in  the  MDPPS  job  aid. 

c.  The  SME  panel  will  recommend  an  algorithm/decision  rule  for  using  the  information 
items,  plus  any  necessary  factor  weightings,  for  using  the  decision  rule. 

2. 1.1. 3  Rationale. 

The  use  of  an  SME  panel  is  essential  to  MDPPS  because  no  objective  method  exists  to  determine  and 
evaluate  the  validity  of  items  us.  as  factors  in  MDPPS’s  decision  rule. 

2. 1.1. 4  Evaluation  Approach. 

The  SME  panel  will  be  selected  and  assembled  to  consensually  evaluate  CRS  information  and  suggest 
how  to  use  this  information  to  identify  passengers  not  needing  special  screening. 

2. 1.1. 5  Analysis  of  MOPs  and  Data  Presentations. 

MOP  1 .  Consensual  evaluation  of  the  validity  for  each  CRS  element. 

MOP  2.  Consensual  evaluation  of  any  additional  information  readily  available  to  the  airline  profiler. 
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TABLE  2.  DATA  AVAILABLE  FROM  NORTHWEST  AIRLINES 


TABLE  REMOVED  FROM  PUBLICATION 


To  obtain  a  copy  of  the  table,  submit  a  written  request  for  the  table,  citing  this  document  and  reason  for 
the  request  to: 

THE  ASSISTANT  ADMINISTRATOR  FOR  CIVIL  AVIATION  SECURITY  ACS-1 
FEDERAL  AVIATION  ADMINISTRATION 
800  INDEPENDENCE  AVENUE,  SW 
WASP’  IGTON,  DC  20591 


TABLE  3.  STRAWMAN  DOMESTIC  PROFILING  DATA  ELEMENTS 


TABLE  REMOVED  FROM  PUBLICATION 


To  obtain  a  copy  of  the  table,  submit  a  written  request  for  the  table,  citing  this  document  and  reason  f.r 
the  request  to: 

THE  ASSISTANT  ADMINISTRATOR  FOR  CIVIL  AVIATION  SECURITY,  ACS-1 
FEDERAL  AVIATION  ADMINISTRATION 
800  INDEPENDENCE  AVENUE,  SW 
WASHINGTON,  DC  20591 
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TABLE  4.  SUBJECT-MATTER  EXPERTS 


Organization 


AMERICAN  AIRLINES  SECURITY 

American  Airlines,  MD-5555  HQs,  4333  Amon  Carter, 

Ft.  Worth,  TX  76155 

CUSTOMS 

Room  4417,  US  Customs  Service,  1301  Constitution  Ave, 
NW  Washington,  DC  20229 

FAA 

ACI-200,  FAA  Headquarters,  800  Independence  Ave,  SW 
Washington,  DC  20591 

FEDERAL  BUREAU  OF  INVESTIGATION  (FBI) 

FBI  Academy,  Quantico,  VA  22135 

IMMIGRATION  AND  NATURALIZATION  SERVICE  (INS) 
1212  Princess  St.,  Alexandria,  VA  22314 

NORTHWEST  AIRLINES  SECURITY 

Northwest  Airlines,  Dept  A4420,  5101  Northwest  Dr. 

St.  Paul,  MN  55111 

UNITED  AIRLINES  SECURITY 

UAL  EXOVS,  POB  66100,  Chicago,  IL  60666 


Falcilitators: 

FAATC 

Aviation  Security  Research  and  Development  Service,  Building  315,  Atlantic  City  International 
Airport,  NJ  08405 

GALAXY  SCffiNTMC  CORPORATION 

2500  English  Creek  Avenue,  Building  11,  Pleasantville,  NJ  08232 
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MOP  3.  Consensual  evaluation  of  weighting  factors  for  information  elements. 

MOP  4.  Consensual  evaluation  of  the  decision  rule  for  combining  information  elements. 

MOP  5.  Job  aid  worksheet  for  profilers. 

2. 1.1. 6  Analysis  Methodology  and  Data  Presentations. 

a.  SME  evaluations  will  be  tabulated  as  shown  in  table  S. 

b.  Each  evaluation  will  represent  the  mean  score  per  information  item. 

c.  Scoring  algorithm(s)  will  be  shown  along  with  directions  for  using  the  decision  aid. 

d.  All  SME  panel  data  will  be  word  processed  as  an  interim  product  (WP  5.0  or  5.1)  in  both 

hard  copy  and  floppy  disk. 

2. 1 . 1 .7  Data  Requirements. 

a.  SME  validity  judgements  for  CRS  information  elements. 

b.  SME  evaluations  of  any  additional  data  elements  to  be  used. 

c.  SME  factor  weights. 

d.  SME  evaluation  of  scoring  algorithm. 

e.  Decision  aid  worksheet 
2.1.2  Issue  2  -  Percentage  Eliminated. 

Does  the  profiling  system  eliminate  most  passengers  from  requiring  special  security  treatment? 

2. 1.2.1  Scope. 

The  aim  of  the  MDPPS  is  to  identify  those  passengers  that  do  not  require  special  security  treatment. 
It  is  presumed  that  the  great  majority  of  passengers  will  fall  into  this  category  and  the  MDPPS  should 
accordingly  eliminate  most  passengers.  The  preliminary  MDPPS  developed  will  be  field  tested  to 
determine  if  an  acceptable  percentage  of  passengers  are  eliminated  from  further  special  treatment. 

2. 1.2.2  Criterion. 

None.  This  issue  is  investigative  in  nature. 
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TABLE  5.  DATA  ELEMENT  EVALUATION 


Profiling  Consensual  Consensual 

Element  Source  Validity  Weight 


2. 1.2.3  Rationale. 

A  study  to  determine  the  percentage  of  passengers  eliminated  from  requiring  additional  special  scrutiny 
is  necessary  to  evaluate  the  practical  effectiveness  of  the  MDPPS.  No  basis  exists  for  selecting  minimal 
clearance  rates. 

2. 1.2.4  Evaluation  Approach. 

A  research  team  will  develop  a  preliminary  MDPPS  and  field  test  to  provide  information  to  ACS  for 
their  determination  of  whether  the  percentage  of  passengers  eliminated  is  acceptable. 

2. 1.2.5  Analysis  of  MQPs  and  Data  Presentation. 

MOP  6.  Number  of  domestic  passengers  profiled  on  various  types  of  flights. 

MOP  7.  Number  of  domestic  passengers  cleared  by  MDPPS. 

2. 1.2.6  Analysis  Methodology  and  Data  Presentations. 

a.  The  total  number  of  passengers  eliminated  from  requiring  additional  special  treatment  will 
be  examined  as  a  proportion  of  the  total  number  of  passengers  profiled. 

b.  Data  will  be  tabulated  as  shown  in  table  6. 

2. 1.2.7  Data  Requirements. 

a.  Number  of  passengers  profiled. 

b.  Number  of  passengers  eliminated. 

c.  Percentage  of  passengers  cleared. 

2.2  OPERATIONAL  SUITABILITY. 

Operational  suitability  will  be  evaluated  by  considering  resource  requirements. 


TABLE  6.  PASSENGER  PROFILING 


Total  Number  of  Total  Number  of  Percentage  of 

Passengers  Profiled  Passengers  Eliminated  Passengers  Eliminated 

2.2.1  Issue  3  -  Resource  Requirements. 

Does  the  domestic  passenger  profiling  system  require  excessive  resource  requirements? 

2.2. 1.1  Scope. 

The  MDPPS  is  intended  for  potential  implementation  on  all  domestic  flights.  Airlines  have  limited 
financial  and  personnel  resources  and  the  MDPPS  should  not  unduly  impact  their  resources  by  requiring 
excessive  monetary  and/or  personnel  commitments. 

2.2. 1.2  Criterion. 

None.  This  issue  is  investigative  in  nature. 

2.2. 1.3  Rationale. 

MDPPS  implementation  will  directly  affect  resource  allocation  within  airlines.  The  field  test  and 
evaluation  of  the  MDPPS  provides  an  opportunity  to  collect  data  regarding  anticipated  resource 
requirements  and  expenditures  involved  in  supporting  the  system. 

2.2. 1.4  Evaluation  Approach. 

A  research  team  will  collect  data  on  the  resource  requirements  for  conducting  objective  MDPPS. 

2.2. 1.5  Analysis  of  MOPs  and  Data  Presentation. 

MOP  8.  Profile  resource  requirements  for  CRS  data. 

2.2. 1.6  Analysis  Methodology  and  Data  Presentations. 

a.  Resource  requirements  necessary  to  implement  and  support  objective  profiling  will  be 
determined. 

b.  Resource  requirements  will  be  tabulated  as  shown  in  table  7. 

2.2. 1.7  Data  Requirements. 

a.  Training  costs  to  prepare  profiler. 
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b.  Labor  cost  for  profiling. 

c.  Miscellaneous  equipment  required. 


2.3  DATA  SOURCE  MATRIX. 

Table  8  shows  the  primary  and  secondary  sources  for  all  data  elements  to  be  collected. 

2.3.1  Issue  1. 

Data  elements  collected  to  determine  if  the  airlines’  CRS  databases  contain  informati  t  can  be  used 
to  profile  passengers  will  be  primarily  provided  by  the  SME  panel.  The  SME  p  will  evaluate 
information  currently  collected  by  airline  personnel  to  determine  whether  it  can  reliably  be  used  to 
identify  domestic  passengers  not  needing  additional  special  security  screening.  Members  of  the  panel 
will  provide  validity  judgments  for  CRS  information,  recommendations  of  any  additional  information 
that  could  be  used  to  profile  passengers,  and  relative  weighting  for  these  profiling  factors.  In  addition, 
the  panel  will  discuss  and  come  to  a  consensus  on  a  scoring  algorithm  for  the  MDPPS. 

TABLE  7.  PROFILING  RESOURCE  REQUIREMENTS 


Personnel 


Trainers 

Profilers 


Equipment 


MDPPS  Instrument 

Clipboards,  Pen/Pencils,  Calculators 

Computer  Terminals 


TABLES.  DATA  SOURCE  MATRIX 


Data  Element 


SME  lOT&E  NW  Airlines 


Validity  Judgements  for  P 

Information  Elements 

Additional  Data  Element  S 

Factor  Weights  P 


P 
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TABLE  8.  DATA  SOURCE  MATRDC  (Continued) 


Data  Element 

SME 

lOT&E 

NW  Airlines 

Evaluation  of  Scoring  Rule 

P 

Number  Profiled 

P 

Number  Cleared 

P 

Percentage  Cleared 

P 

Training  Costs 

S 

P 

Labor  Costs 

S 

P 

Northwest  will  also  provide  data  elements  regarding  Issue  1  along  with  critical  profiling  items  (i.e.,  items 
that  automatically  clear  or  fail  to  clear). 

2.3.2  Issue  2. 

Data  elements  collected  to  determine  if  the  MDPPS  developed  will  eliminate  most  passengers  from 
requiring  additional  special  security  treatment  will  be  collected  during  Initial  Operational  Test  and 
Evaluation  (lOT&E).  The  percentage  of  passengers  cleared  will  be  obtained  by  dividing  the  number  of 
passengers  eliminated  by  the  number  of  passengers  profiled. 

2.3.3  Issue  3. 

Data  elements  to  support  the  FAA’s  decision  on  whether  the  MDPPS  requires  excessive  resource 
requirements  will  be  collected  from  the  airlines.  The  perceived,  estimated,  and  actual  time  and  activity 
commitments  required  to  implement  and  maintain  the  system  will  be  gathered  and  presented  to  the 
airline  representatives.  A  cost-benefit  analysis  will  be  conducted  between  the  test  team  and  Northwest. 

2.4  TEST  APPROACH. 

2.4.1  Test  Scone. 

The  scope  of  this  test  includes  those  activities  necessary  to  identify  proflling  criteria,  create  a  manual 
profiling  form,  and  conduct  an  operational  test  to  determine  the  feasibility  of  the  system. 
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2.4.2  Factors  and  Conditions. 

Table  9  lists  the  factors  and  conditions,  for  potentially  relevant  variables  including  subject  variables,  and 
their  control  procedures. 

2.4.3  Sample  Size  and  Other  Considerations. 

The  SME  panel  will  consist  of  no  more  than  ten  individuals  representing  various  government  and  airline 
agencies  responsible  for  security,  safety,  and  legal  issues  involving  air  travel,  passenger  safety,  and  anti¬ 
terrorism.  Northwest  will  provide  the  necessary  computer  equipment  to  gather  all  CRS  data  and  ticket 
lifts,  and  test  personnel  (Galaxy/Northwest)  will  perform  the  manual  profiling.  The  focus  of  the  effort 
is  to  get  a  percentage  of  "known"  versus  "unknown"  passengers  using  the  indicators  identified  by  the 
SME  panel.  Passengers  will  be  randomly  selected  for  on-line  profiling  and  entire  flights  will  be  profiled 
off-line.  The  total  number  of  passengers  profiled  will  be  approximately  500. 

2.5  EVALUATION  DATABASE  STRUCTURE. 

2.5.1  Objective  and  Variable  Data. 

Table  5  shows  the  layout  for  data  collected  on  profiling  elements  as  well  as  their  validity  and  weighting. 

2.5.2  Profiling  Data. 

Table  10  shows  the  layout  for  data  collected  during  lOTifcE  regarding  passengers  eliminated  by  the 
MDPPS. 

2.5.3  Cost  Data. 

Table  11  shows  the  layout  for  data  collected  during  the  panel  discussions,  lOT&E,  and  ensuing 
discussions  regarding  the  number  of  passengers  eliminated  by  the  MDPPS. 

TABLE  9.  FACTORS  AND  CONDITIONS 


Factor  Control 

Flight  Factors 

Systematic  Variation 
Systematic  Variation 
Uncontrolled 
Uncontrolled 


Route 

Time  of  day 
Day  of  week 
Time  of  year 
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TABLE  9.  FACTORS  AND  CONDITIONS  (Continued) 


Factor 

Control 

Passenger  Factors 

Age 

Random  Selection 

Gender 

Random  Selection 

Occupation 

Random  Selection 

Travel  purpose 

Random  Selection 

Profiler  Factors 

Years  on  job 

Held  Constant 

Positions  held 

Held  Constant 

Training  Courses/Education 

Held  Constant 

Past  job  performance 

Held  Constant 

Attitude  toward  job 

Held  Constant 

TABLE  10.  PASSENGER  PROFILES  DATABASE  STRUCTURE 

Flight  Number 

Route 

Number  Profiled 

Number  Cleared 

Percentage  Cleared 

Average  Profile  Time 

Standard  Deviation 
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TABLE  11.  RESOURCE  REQUIREMENTS  DATABASE  STRUCTURE 


Number  Trained 
Training  Time 
Training  Cost 
Number  Profiling 
Profiling  Time 
ProEling  Cost 


3.  TEST  DESIGN. 

3.1  TEST  CONCEPT. 

3.1.1  Introduction. 

The  MDPPS  is  a  proposed  tool  for  passive  screening.  Background  data  on  passengers  will  be  collected 
and  assessed  as  to  whether  the  passenger  represents  a  "known"  entity  to  the  airline.  Passengers  who  are 
determined  to  represent  at  least  a  minimally  known  entity  will  be  passed  through  subsequent  in-depth 
security  checks.  Passengers  who  do  not  meet  the  criteria  will  be  subjected  to  special  security  checks, 
possibly  including  active  profiling,  passenger-bag  matching,  and/or  baggage  screening  through  Explosive 
Detection  Systems  (EDS). 

3.1.2  Operational  Context. 

The  MDPPS  will  perform  its  mission  at  airport  check-in  points.  Eventual  implementation  of  the  MDPPS 
into  the  operations  of  the  airport  will  be  dictated  by  perceived  threat  levels  across  the  aviation  system. 

3.1.3  Test  Phases. 

The  lOT&E  will  be  conducted  in  three  phases:  Phase  1  -  Training,  Phase  2  -  Pilot  Testing,  and  Phase 
3  -  Operational  Testing. 

3.1.4  Test  Unit  Configuration. 

Test  personnel  will  be  Northwest  and  Galaxy  Scientific  employees.  Other  participants  will  be  members 
of  the  SME  panel  identified  in  section  2. 
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3.1.5  Training  Concept. 


3. 1.5.1  Training  of  Test  Players. 

Training  for  the  SME  panel  personnel  will  be  minimal  and  conducted  by  FAA  and  Galaxy  Scientific 
personnel  at  the  beginning  of  the  SME  panel  working  session.  Training  of  Northwest  profiling  personnel 
will  be  conducted  by  Galaxy  Scientific  prior  to  the  operational  test  phase.  The  training  will  consist  of 
both  classroom  instruction  and  hands-on  profiling  trials.  The  training  will  be  presented  in  accordance 
with  (lAW)  the  Program  of  Instruction  (POI)  and  lesson  plans  in  appendix  A. 

3. 1.5.2  Training  of  Test  Organization. 

Training  of  Galaxy  Scientific  and  FAA  test  team  personnel  will  occur  prior  to  the  operational  test 
(appendix  A). 

3.1.6  Instrumentation. 

The  major  instrumentation  requirement  is  access  to  Northwest’s  computer  terminals  at  the  Milwaukee 
Airport  and  provisions  for  timing  the  profiler’s  duration  to  complete  each  profile. 

3.1.7  Test  Design  Limitations. 

The  focus  of  the  MDPPS  project  is  to  determine  the  feasibility  of  introducing  the  system  into  the 
operational  environment.  The  scheme  for  introduction  is  to  use  the  MDPPS  tool  at  the  point  of 
passenger  baggage  checking.  However,  testing  at  this  location  would  maximally  disrupt  the  airline. 
Data  will  accordingly  be  collected  at  the  gate  agent  location.  Time  requirements  for  real-time  profiling 
at  the  gate  will  be  extrapolated  to  the  baggage  check  in  site. 

3.2  TEST  DETAILS. 

3.2.1  Issue  1. 

Do  airlines  CRS  databases  contain  information  elements  appropriate  for  judging  passengers  not  to  be 
a  threat  on  domestic  flights? 

3.2. 1.1  General  Methodology. 

a.  A  SME  Workshop  will  accomplish  Task  3  of  Phase  1 .  This  workshop  will  be  conducted 
in  the  Washington,  D.C.  area  and  use  a  Delphi  technique  for  its  methodology.  The  Delphi  literature 
provides  a  complete  blueprint  for  the  construction  and  execution  of  the  SME  workshop  and  is  abstracted 
in  appendix  C.  The  Contracting  Officer’s  Technical  Representative  (COTR)  will  be  the  workshop 
facilitator. 


b.  A  pre-workshop  package  (appendix  D)  will  be  sent  to  the  SMEs  approximately  14  days 
prior  to  convening  the  workshop.  These  materials  will  familiarize  the  SMEs  with  the  objectives  and 
requirements  for  the  MDPPS.  In  addition,  SMEs  will  be  provided  with  listings  of  CRS  data  elements. 
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FAA  sample  profiling  criteria,  and  other  applicable  data.  SMEs  will  be  asked  to  return  portions  of  the 
pre-workshop  package,  prior  to  convening,  for  initial  data  analysis. 

c.  Data  collection  will  be  conducted  at  Northwest  in  the  Milwaukee,  Wisconsin  terminal. 
A  random  sample  of  passenger  CRS  records  will  be  examined  by  ticket  agents  with  real-time  computer 
access  to  CRS  data.  These  profiles  will  be  done  by  viewing  data  elements  during  passenger  interactions. 
The  duration  of  this  passenger  profihng  will  be  measured.  In  addition,  entire  flights  will  be  profiled  off¬ 
line  by  examining  the  CRS  printout.  This  technique  will  also  be  timed. 

3.2. 1.2 _ Data  Requirements. 

Data  requirements  include  the  following: 

a.  A  complete  listing  of  the  data  elements  available  in  the  Northwest’s  CRS  computer 
system,  including  the  PNR,  the  frequent  flyer  database,  and  the  Baggage  Tracking  database. 

b.  Candidate  objective  screening  criteria  obtained  from  SME  panel  recommendations. 

c.  A  weighting/decision  system  for  the  screening  criteria  will  be  derived  from  the  SME 

panel. 

d.  A  random  sample  of  CRS  database  records  (PNR,  Frequent  Flyer,  Baggage)  for  real-time 
and  complete  flight  CRS  records  for  off-line  processing. 

e.  Field  observations  of  ticket  agent-passenger  interactions,  including  screen  captures  of 
computer  data  by  the  ticket  agent  and  off-line  evaluations  of  preprinted  passenger  data. 

OTMOP  1.  Consensual  evaluation  of  the  validity  for  each  CRS  element. 

The  SMEs  will  receive  the  available  CRS  data  elements  in  the  pre-meeting  package  mailed  to  each 
participant.  Their  review  and  ranking  will  be  provided  on  attached  forms  and  returned  to  the  test  team 
for  initial  analysis.  The  SME  panel  will  then  arrive  at  a  consensus  during  its  workshop  on  profile  data 
elements,  through  the  modified  Delphi  procedure. 

OTMOP  2.  Consensual  evaluation  of  any  additional  information  readily  available  to  the  airline  profiler. 

The  SMEs  will  suggest  additional  data  elements  to  be  included  in  the  passenger  profile  through  the  pre¬ 
meeting  responses  and  the  modified  Delphi  procedure  at  the  meeting. 

OTMOP  3.  Consensual  evaluation  of  weighting  factors  for  information  elements. 

The  SMEs  will  provide  the  importance  of  weighting  data  elements  through  the  modified  Delphi 
technique  utilized  at  the  SME  meeting. 
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OTMOP  4.  Consensual  evaluation  of  the  decision  rule  for  combining  information  elements. 


The  SME  panel  will  formulate  a  decision  rule  for  combining  the  profile  data  elements  into  a  tool  to 
screen  passengers.  The  concept  for  the  decision  rule  is  to  provide  a  score  for  each  passenger  based  on 
the  find  set  of  data  elements  chosen  from  the  CRS  databases  and  the  weights  derived  in  OTMOP  3. 
Passengers  achieving  a  minimum  score  would  be  passed  on  in  the  security  system  without  further  special 
security  efforts.  The  formulation  of  the  decision  rule  will  be  elicited  during  the  SME  panel  meeting 
using  the  modified  Delphi  technique. 

OTMOP  5.  Job  aid  worksheet  for  profilers. 

Based  on  the  recommendations  of  the  SME  panel  for  data  elements,  relative  weightings,  and  the  decision 
rule,  a  job  aid  worksheet  will  be  constructed.  This  worksheet  will  provide  the  tool  to  perform  the 
manual  profiling  to  be  tested  for  Issues  2  and  3.  The  construction  of  the  job  aid  worksheet  will  be 
performed  by  the  Galaxy  Scientific  test  team.  Acceptance  of  the  job  aid  worksheet  will  be  by  consensus 
of  the  FAA  monitors  and  the  Northwest  corporate  security  personnel. 

3.2.2  Issue  2. 

Does  the  profiling  system  eliminate  most  passengers  from  requiring  special  security  treatment? 

3.2.2. 1  _ General  Methodology. 

a.  The  CRS  and  ticket  lift  data  will  be  gathered  at  Northwest’s  Milwaukee  terminal  and 
manual  profiling  performed  using  a  combination  of  Northwest  ticket  agents  and  Galaxy  Scientific  test 
team  members. 

b.  Historical  terrorist  or  threat  profiles  will  be  inserted  into  two  typical  flights  to  determine 
if  the  threat  subjects  are  passed  through  the  manual  profiling  system. 

3.2.2.2  _ General  Data  Requirements. 

Data  requirements  include  the  following: 

a.  CRS  and  ticket  lift  data  for  a  total  of  approximately  500  randomly  selected  passengers  and 
entire  flights  at  the  Milwaukee  airport. 

b.  Completed  job  aid  worksheets. 

c.  Historical  terrorist  or  threat  profiles  from  the  University  of  Louisville  (if  available  in  time). 

OTMOP  6.  Number  of  domestic  passengers  profiled  on  various  types  of  flights. 

Manual  on-line  profiling  will  be  conducted  on  a  random  selection  of  passengers  from  numerous 
Northwest.  Some  flights  will  have  all  passengers  profiled  off-line. 
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OTMOP  7.  Number  of  domestic  passengers  cleared  by  MDPPS. 


The  cleared  and  not-cleared  rates  will  be  determined.  Passengers  not-cleared  will  be  categorized  as  due 
to  insufficient  information  available  in  the  file  or  as  a  failure  to  clear  on  merit. 

3.2.3  Issue  3. 

Does  the  domestic  passenger  profiling  system  require  excessive  resource  requirements? 

3.2.3. 1  _ General  Methodology. 

a.  Collect  cost  estimates,  from  Northwest  auditors  and  security  personnel,  using  the  data 
collected  in  Issue  2. 

b.  Perform  a  cost-benefit  analysis  using  the  provided  cost  estimates. 

3.2.3.2  _ General  Data  Requirements. 

Data  requirements  include  the  following: 

a.  Profiling  training  times  derived  in  Issue  2. 

b.  Profiling  execution  times  per  passenger  and  per  flight  derived  in  Issue  2. 

c.  Northwest’s  cost  estimates  for  personnel,  training,  and  resource  requirements  based  on  the 
data  in  a  and  b. 

OTMOP  8.  Profile  resource  requirements  for  CRS  data. 

Using  the  time  to  profile  and  training  data  collected  in  Issue  1,  Northwest  will  provide  additional  staffing 
estimates.  Included  in  these  estimates  will  be  items  concerning  additional  airline  profilers,  as  well  as 
indirect  training  personnel  and  administrative  support.  A  cost-benefit  analysis  will  be  conducted  using 
the  Northwest’s  estimates  on  direct  and  indirect  costs,  time  required,  and  percentage  of  passengers 
cleared. 
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APPENDIX  A 


i 

I 

TRAINING  CONCEPT 


1.  SME  PANEL  TRAINING. 

Dr.  Lofaro  will  provide  the  SME  panel  with  approximately  45  minutes  to  one  hour  of  instruction  on  the 
following: 

a.  group  processes  and  products 

b.  consensus 

c.  "group  think." 

Included  in  this  training  will  be; 

a.  a  division  of  the  workshop  participants  into  two  groups 

b.  an  examination  and  discussion  of  the  workshop  objectives 

c.  an  exposition  of  the  workshop  methodology 

d.  an  examination  of  the  worksheet  results. 

2.  GALAXY  SCIENTinC  PERSONNEL  TRAINING. 


Nor  thwest  personnel  will  train  Galaxy  Scientific  personnel  on  the  meaning  of  the  data  elements  in  the 
PNR  and  navigation  through  the  PNR  database.  An  exercise  will  be  conducted  to  permit  Galaxy 
personnel  to  practice  and  become  familiar  with  the  system. 

3.  NORTHWEST  AIRLINES  TRAINING. 

Galaxy  Scientific  personnel  will  brief  Northwest  ticket  and/or  gate  agents  about  the  MDPPS  project. 
Following  this.  Galaxy  Scientific  and  Northwest  personnel  will  provide  training  on  the  use  of  the 
MDPPS  instrument,  the  meaning  of  the  data  elements  in  the  PNR,  and  navigation  through  the  PNR 
database.  An  exercise  will  be  conducted  to  permit  North. zest’s  agents  to  practice  and  become  familiar 
with  the  MDPPS  instrument  and  the  PNR  system. 

This  training  will  be  interactive,  with  suggestions  from  the  Northwest’s  agents  being  discussed  and  the 
procedures  modified  appropriately. 
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APPENDIX  C 


THE  DELPHI  TECHNIQUE 


Delphi  techniques  have  become  common  methodologies  for  eliciting  group  analyses,  expert  opinions, 
and  evaluations  on  a  variety  of  topics.  In  general,  standard  Delphi  techniques  include  anonymity  of 
respondent,  multiple  iterations,  convergence  of  the  distribution  of  answers,  and  a  statistical  group 
response  (e.g.,  median  and  interquartile  range)  preserving  intact  a  distribution  that  still  may  remain  wide 
(Judd,  1972).  Lofaro  developed  modifications  for  small-group  Delphi  processes  and  facilitated  this 
paradigm  in  FAA  sponsored,  contractor  delivered  workshops  (aircrew  training  and  assessment; 
aeronautical  decision  making;  ATCS  selection).  As  a  result  of  these  Delphi  workshops,  traditional 
Delphi  processes  were  modified  into  a  new  paradigm  for  small-scale  Delphi  projects.  The  paradigm 
consists  of  the  following  (Lofaro,  1992): 

a.  Formal  instruction  in  group  processes,  group  dynamics,  and  methods  of  consensus.  This 
includes  guided  exercises  in  group  consensus  followed  by  evaluation  and  critique  of  the  group  techniques 
by  group  members  and  the  facilitator. 

b.  The  use  of  an  iterative,  step-wise  process  to  achieve  the  Delphi  objectives.  This  process 
is: 


1.  Anonymous  individual  ratings 

2.  Sub-group  discussions 

3.  Iterated  individual  ratings 

4.  Sub-group  discussions  and  consensual  ratings  (possible  iterations) 

5.  Intact  group  discussions,  with  possible  iterated  ratings,  to  achieve  a  final,  group 
consensus  on  the  ratings. 

c.  The  use  of  a  procedure  in  which  the  database  for  each  step  in  an  objective  evolves  from 
the  preceding  steps.  This  is  done  through  computer  analyses  and  reprinting  the  previous  steps  with  the 
new  data  incorporated.  The  delay  that  occurs  between  iterations,  while  the  data  analyses  are  run  and 
new  worksheets  are  printed,  is  used  to  allow  participants  to  proceed  to  another  objective  or  task.  This 
work  schema  also  precludes,  to  some  extent,  the  "let’s  get  it  done"  syndrome,  which  often  develops 
when  the  same  data  are  iterated  and  reiterated. 
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INTRODUCTION 


Terrorists  are  expected  to  go  out  of  their  way  to  remain  anonymous  and  our  approach  to  passenger 
profiling  accordingly  emphasizes  "knowing"  the  passenger.  Airlines  have  travel  information  for  each 
passenger  that  varies  qualitatively  and  quantitatively.  We  are  asking  the  SMEs  to  help  develop  a 
profiling  procedure  which  uses  critical  variables  and  indicates  whether  enough  is  known  about  the 
traveler.  "Known"  passengers  are  then  cleared  from  special  security  provisions.  This  approach  contrasts 
with  one  attempting  to  identify  threat  individuals. 

To  help  us  determine  the  appropriate  objective  data  elements  and  weighting  criteria,  we  have  invited 
several  individuals  from  a  number  of  organizations  concerned  with  security.  Representatives  from  the 
Federal  Bureau  of  Investigations  (FBI),  the  U.S.  Customs  Service,  the  Immigration  and  Naturalization 
Service  (INS),  and  several  airline  security  divisions  are  participating  in  the  workshop. 

A  modification  of  a  measurement  method  known  as  the  Delphi  technique  will  be  used  to  collect  data 
during  the  SME  workshop.  In  this  data  collection  technique,  subject-matter  experts  provide  information 
individually  prior  to  attending  a  group  meeting.  Each  expen’s  responses  are  analyzed  and  then  averaged 
by  the  monitoring  team.  The  compiled  information  is  then  presented  to  the  expens  at  a  group  meeting, 
or  workshop,  and  further  discussed  to  clarify  the  information  and  provide  group  consensus. 

To  make  the  most  efficient  use  of  our  one-day  workshop  time,  we  are  asking  you  to  review  the  enclosed 
information  and  complete  the  worksheets.  We  request,  if  possible,  that  you  use  the  enclosed  pre-paid 
envelope  to  mail  us  the  completed  worksheets  no  later  than  February  8, 1994.  This  way  we  can  compile 
the  individual  data  prior  to  the  meeting,  and  thus  make  the  best  use  of  the  workshop  time. 

The  Objectives  and  Worksheets  sections  of  this  briefing  describe  in  greater  detail  the  objectives  of  the 
panel  discussions  and  the  activities  we  are  asking  you  to  perform  both  prior  to  arriving  and  during  the 
workshop. 
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AGENDA 


Location:  Galaxy  Scientific  Corporation  Date:  February  14,  1994 

4900  Seminary  Road 
Suite  530 

Alexandria,  VA  22311 
(703)379-2107 

Introductions  8:00  -  8:30 

Brief  Discussion  and  Explanation  of  Objectives  and  Methodology 

Discussion  for  Objective  1 

BREAK 

Summary  of  Conclusions  from  Discussion  for  Objective  1 
LUNCH  12:00  -  1:00 
Discussion  for  Objective  2 
BREAK 

Summary  of  Conclusions  from  Discussion  for  Objective  2 
Workshop  Conclusions 
Final  Remarks  4:00 


D-4 


LIST  OF  PARTICIPATING  ORGANIZATIONS 


SUBJECT-MATTER  EXPERTS 

FBI  Academy 
Quantico,  VA  22135 

United  Airlines 
UAL.  EXOVS 
POB  66100 
Chicago,  IL  60666 

Immigration  and  Naturalization  Service 
1212  Princess  St. 

Alexandria,  VA  22314 

Northwest  Airlines 
Dept.  A4420 
5101  Northwest  Dr. 

St.  Paul,  MN  55111 

American  Airlines 
MD-5555  HQs 
4333  Amon  Carter 
Ft.  Worth.  TX  76155 

FAA  Headquarters,  ACI-200 
800  Independence  Ave.,  SW 
Washington,  DC  20591 

U.S.  Customs  Service 
Room  4413 

1301  Constitution  Ave.,  NW 
Washington,  DC  20229 

PANEL  FACILITATORS 


Aviation  Security  R&D  Service 

FAA  Technical  Center 

Atlantic  City  International  Airport,  NJ  08405 

Galaxy  Scientific  Corporation 
2500  English  Creek  Ave.,  Building  1 1 
Pleasantville,  NJ  08232 
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OBJECTIVES 


OBJECTIVE  1 

Identify  objective  data  elements  for  passenger  profiling. 

A.  Identify  those  data  elements  currently  available  to  airline  personnel  that  are  helpful  in 
passenger  profiling. 

Most  major  airlines  use  a  computer  based  passenger  reservation  system.  These  systems  collect 
information  about  a  passenger  and  their  flight  activities.  It  is  thought  that  the  information  collected  by 
such  systems  may  be  used  to  determine  that  a  potential  passenger  does  not  require  additional  security 
attention.  The  SMEs  will  review  a  list  of  this  information  and  determine  which  elements,  if  any,  are 
appropriate  for  passenger  profiling,  and  rate  their  criticality  in  determining  a  non-threat  passenger.  A 
worksheet  for  this  objective  is  provided. 

B.  Identify  additional  data  elements  that  should  be  included  in  a  passenger  profiling  system. 

Although  the  airlines  data  collection  systems  record  a  great  deal  of  information  about  a  passenger,  not 
all  critical  factors  will  be  included,  llie  SMEs  will  be  asked  for  recommendations  on  additional  data 
elements  that  should  be  included  in  passenger  profiling,  but  are  not  currently  collected  by  the  airlines. 
A  worksheet  for  this  objective  is  provided. 

OBJECTIVE  2 


Determine  a  weighting  mechanism  and  decision  rule  for  the  objective  data  elements  identified  as 
necessary  for  passenger  profiling. 

Each  piece  of  information  collected  about  a  potential  passenger  does  not  have  the  same  degree  of 
importance  in  determining  whether  or  not  a  passenger  requires  additional  special  security  screening.  For 
example,  frequent  flyers  or  travel  companions  may  be  seen  as  more  indicative  of  a  non-threat  potential 
than  city  of  origin  and  method  of  payment. 

To  determine  an  appropriate  weight  for  each  identified  data  element,  we  will  ask  the  SMEs  to  achieve 
a  consensus  on  a  weighting  mechanism  and  decision  tool.  The  goal  of  the  MDPPS  is  to  provide  each 
passenger  with  a  "score"  that  will  enable  a  definitive  decision  to  be  made  as  to  the  necessity  of  requiring 
additional  special  security  screening.  The  decision  tool  for  the  MDPPS  is  a  direct  result  of  the  weighting 
mechanism  chosen.  The  scoring  algorithm  is  dependent  on  the  type  of  numerical  data  supplied  by  the 
profiling  system.  As  such,  these  two  items,  the  weighting  mechanism  and  the  decision  tool,  will  be 
developed  in  tandem.  Once  the  weighting  mechanism  and  decision  tool  has  been  agreed  upon,  it  will 
be  applied  to  each  of  the  identified  objective  data  elements.  No  worksheet  for  this  objective  is  provided, 
but  please  put  thought  to  this  issue  and  come  prepared  for  discussion. 
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WORKSHEETS 


WORKSHEET  1  -  Objective  1 

A.  Identify  those  data  elements  currently  available  that  are  helpful  in  passenger  profiling. 

Worksheet  I  provides  a  list  of  the  type  of  information  collected  by  currently  used  passenger  reservation 
or  potential  profiling  systems.  Using  the  criticality  rating  scale  provided,  please  rate  all  items  as  to  their 

value  in  passenger  profiling. 


WORKSHEET  REMOVED  FROM 
PUBLICATION 


To  obtain  a  copy  of  the  worksheet,  submit  a  written  request  for  the  worksheet,  citing  this  document  and 
reason  for  the  request  to; 

THE  ASSISTANT  ADMINISTRATOR  FOR  CIVIL  AVIATION  SECURITY,  ACS-1 
FEDERAL  AVIATION  ADMINISTRATION 
800  INDEPENDENCE  AVENUE,  SW 
WASHINGTON,  DC  20591 
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WORKSHEET  2  -  Objective  1 

B.  Identify  additional  data  elements  that  should  be  included  in  a  passenger  profiling  system. 

Please  indicate  any  additional  data  elements  you  feel  should  be  included  in  a  passenger  profiling 
system. 


